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Abstract 


This study investigated flipped learners’ perceptions, perceived usefulness, intention to register for 
flipped learning classes, and learning engagement using cluster analysis with a sample of 306 
undergraduate students in flipped classes. The students were classified into five clusters based on their 
level of social influence and cognitive instrumental processes using the technology acceptance model 2. 
There were significant differences among the five clusters in perceived usefulness, intention to register 
for flipped learning classes, and learning engagement. The perception of and preference for flipped 
learning also varied by cluster. The flipped learning feature that most participants commented on was 
online pre-learning sessions. The advantages of flipped learning that have to do with online pre-learning 
sessions include facilitating understanding of lectures, providing review opportunities for learning, flexible 
learning time, and individualized learning. These advantages indicate that providing students with 
instructionally sound pre-learning sessions leads to successful flipped learning. The research findings also 
suggest that flipped class instructors and school administrators utilize cluster analysis with meaningful 
variables to provide students with effective and tailored learning support. 


Keywords: flipped learning, perceptions of flipped learning, perceived usefulness, learning engagement, 
registration intention, cluster analysis 


INTRODUCTION 


Flipped learning has become an increasingly popular option in higher education (Johnson et al., 2015; Lee & 
Choi, 2019; Steen-Utheim & Foldnes, 2018). The rapid growth of flipped learning is due to its strengths as an 
instructional approach, such as improved learning engagement, cost-efficiency of implementing learner- 
centered approaches to a large number of students, and increased learning outcomes (Al-Samarraie et al., 
2019; Bergmann & Sams, 2012; Fulton, 2012). He et al. (2016) described three components of flipped 
learning: “Flipped classrooms should feature (1) mandatory pre-class learning of new material followed by 
(2) in-depth explanation, practice, and productive use of knowledge in class through active learning 
techniques, where (3) class attendance is mandatory” (p. 61). Goodwin and Miller (2013), and O’Flaherty and 
Phillips (2015) also explained that flipped learning is the best practice of blended learning, which has recently 
become popular in higher education. It has recently become even more popular as institutions have shifted 
to online formats due to COVID-19 pandemic restrictions. 


Flipped learning requires students to complete pre-learning sessions prior to class and then participate in 
engaging learner-centered activities in class. Lee and Choi (2019) emphasized the significance of pre-learning 
sessions for effective flipped learning courses. They found that the influence of pre-learning sessions on the 
success of learning outcomes was approximately twice as large as outcomes with in-class learning. On the 


Copyright © 2022 by the authors; licensee CEDTECH by Bastas. This article is published under the terms of the 
Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/). 


Doo / Contemporary Educational Technology, 2022, 14(1), ep331 


negative side, however, He et al. (2016) pointed out that non-compliance with watching or reading materials 
for pre-learning sessions is a hindrance for flipped learning. This barrier indicates that the roles and 
responsibilities of students in flipped learning classes are considerably different than in traditional lecture- 
based classes. The requirements and expectations of students in flipped learning classes are also quite 
different from how students have been used to learning (Ng, 2018). Koh (2019) found that students perceived 
that flipped classes require more time to learn the pre-learning materials and reported having a heavier 
workload. For these reasons, students’ perceptions of flipped learning and their motivations for choosing 
flipped learning classes may be different from the perspectives of instructors or school administrators. 


As more students have recently enrolled in flipped learning classes in higher education, the perceptions and 
needs of students for flipped learning have become more diverse (Forsey et al., 2013; Hao, 2016). Based on 
their scoping review of flipped learning, O’Flaherty and Phillips (2015) reported that students’ perceptions of 
flipped learning were largely positive; however, they also identified negative perspectives of flipped learning. 
Hao (2016) examined undergraduate students’ perceptions of flipped learning and concluded that flipped 
learning cannot be applied to “all students, all contexts, and all disciplines.” While substantial research on 
flipped learning has examined the effectiveness and efficiency of flipped learning, including Forsey et al. 
(2013), Hao (2016), He et al. (2016), O’Flaherty and Phillips (2015), Nouri (2016), and Wang and Zhu (2019), 
few studies have explored students’ perceptions of and intention to register for flipped learning classes with 
the segmentation of learners. The segmentation of flipped learners is expected to enable instructors or 
school administrators to apply effective instructional strategies and provide appropriate learning support in 
flipped learning. Given that there is a wide range of students’ perceptions of and needs for flipped learning, 
this study classified flipped learners into meaningful groups using cluster analysis by applying the extension 
of the technology acceptance model (TAM2) (Venkatesh & Davis, 2000). 


TAM2 is an extension of Davis’ (1989) technology acceptance model developed by adding two external 
processes (i.e., social influence and cognitive instrumental processes) to increase its explanatory power. The 
purpose of this study is to understand flipped learners’ perceptions, perceived use, intention to register for 
flipped learning classes, and learning engagement using cluster analysis. We posed two research questions 
to guide the study: 


1. Do perceived usefulness, intention to register for flipped learning classes, and learning engagement of 
flipped learners differ among groups with different levels of social influence and cognitive instrumental 
processes? 


2. What are students’ perceptions of and preferences for flipped learning? What do students like the most 
and the least about flipped learning? 


LITERATURE REVIEW 


Students’ Perceptions of Flipped Learning 


Flipped learning is defined as “a pedagogical approach in which direct instruction moves from the group 
learning space to the individual learning space, and the resulting group space is transformed into a dynamic, 
interactive learning environment where the educator guides students as they apply concepts and engage 
creatively in the subject matters” (Flip Learning, 2014). Flipped learning has been touted as an effective 
instructional approach from the perspectives of instructors, school administrators, and education 
researchers (Hwang et al., 2019; Steen-Utheim & Foldnes, 2018). McDonald and Smith (2013), and O’ Flaherty 
and Phillips (2015) listed the advantages of flipped learning including “encouraging students to become self- 
directed learners,” “flexibility in learning,” and “promoting ownership of learning.” Koh (2019) conducted an 
extensive literature review and summarized the effectiveness of flipped learning in higher education, 
including (1) personalization, (2) higher-order thinking, (3) collaboration, and (4) self-direction. Steen-Utheim 
and Foldnes (2018) conducted a research investigating the perception of flipped learning from students’ 
perspectives. They examined the affective dimension of learning engagement of flipped learners using 
qualitative data with 12 freshmen at a Norwegian business school who were enrolled in both a flipped class 
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and a traditional lecture-based class. The results of the data analysis yielded the following seven themes 
about flipped learning: (1) commitment to peers, (2) learning with peers, (3) being recognized [as individuals], 
(4) reduced anxiety [or obtaining emotional safety], (5) using videos to learn new content, (6) [different] 
instructor relationship, and (7) physical learning environment [different from traditional lecture-based 
classes]. Based on these research findings, Steen-Utheim and Foldnes (2018) emphasized the importance of 
students’ perceptions of flipped learning and affective learning engagement for successful learning 
outcomes. 


Given that flipped learning is a learner-centered instructional approach and requires students to play a more 
active role in learning, students’ perceptions of and preferences for flipped learning is important for 
successful learning outcomes. To examine students’ perceptions of flipped learning, O’Flaherty and Phillips 
(2015) conducted an extensive review of 28 relevant studies on flipped learning in higher education and 
found that students’ perceptions of flipped learning were largely positive; however, some students had 
negative perspectives of flipped learning. In other words, flipped learning is not applicable or satisfactory for 
all students. Hao (2016) also analyzed students’ perceptions and readiness level for flipped learning with 84 
undergraduate students in Taiwan. Around 60% of the participants were positive about flipped learning in 
general. However, only 39.3% reported that flipped learning met their learning needs and only 44% said that 
they would take flipped courses in the future. What students liked about flipped learning included, 
“recognition of learning needs,” “more interaction in classes,” and “more effective use of class time.” 
However, students disliked the “requirement for self-discipline,” “the requirement of prior knowledge and 
academic motivation,” and that they had “less learner control.” 


Forsey et al. (2013) also surveyed 74 students in a sociology class using flipped learning in Australia and found 
that only slightly more than half (53%) of the students indicated that flipped learning suited their learning 
needs. What aspects the participants valued in flipped learning included “flexibility,” “the richness of the 
learning content,” and “productivity in learning.” The students were also concerned that online pre-learning 
might substantially replace instructor-led lectures. Wang and Zhu (2019) also examined 73 first-year 
students’ perceptions of flipped learning in a chemistry course using MOOC-based flipped learning materials 
in China. Although there were significant learning gains for the flipped learners, student evaluations of the 
overall learning experience were slightly lower than their learning performance. Wang and Zhu found that 
most students liked the student-student interaction and active learning results in class; however, the pre- 
learning materials (i.e., using existing MOOCs) did not satisfy students’ needs. This finding indicates that there 
should be a good fit between the pre-learning materials and class activities. 


Nouri (2016) also compared the effectiveness of flipped learning between high achievers and low achievers 
in a university in Sweden. The participants of their study had positive perceptions of flipped learning as an 
instructional approach in terms of the use of video, flexibility, and mobility, all of which support learning with 
meaningful learning activities. Although no significant differences in the attitudes toward flipped learning 
were found between high achievers and low achievers, the study found that the low achievers appreciated 
the use of video more as pre-online learning materials than the higher achievers. In addition, the perceived 
learning gains of low achievers were higher than the high achievers. Nouri explained that low achievers may 
benefit more from flipped learning due to the individualization (e.g., more opportunities to learn at their own 
pace) and repetition (e.g., ability to pause and rewind the pre-learning online materials). 


He et al. (2016) investigated the effects of flipped learning on learning outcomes and students’ perceptions 
of flipped learning with 676 freshmen using a treatment and comparison group design. They found that 
flipped learning slightly increased students’ final exam performance, but not motivation. He et al. (2016) 
reported that the participants in the treatment group reported mixed feelings about flipped learning. 
Exemplary positive comments about flipped learning included, “learning at one’s own time and pace, better 
preparation for class, and more problem solving and teacher-student interaction” (p. 70). These studies on 
students’ perceptions of flipped learning reveal a wide range of student perceptions and intention to register 
for flipped learning classes. To implement flipped learning more effectively, it is necessary to understand 
learners’ perceptions at a deeper level (e.g., using segmentation or clusters) to help instructors and 
administrators apply customized instructional strategies that address the preferences of each learner cluster. 
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Figure 1. Technology acceptance model 2 
Note: Restructured extension of the technology acceptance model (Venkatesh & Davis, 2000, p. 188) 


Technology Acceptance Model 2 


The technology acceptance model (TAM) (1989) has been used to explain adoption of innovation or new 
technologies in various realms including in education (Abdullah & Ward, 2016; Agudo-Peregrina et al., 2014; 
Chen Hsieh et al., 2017; Park et al., 2012). TAM explains that two critical elements, perceived usefulness and 
perceived ease of use, largely determine individuals’ technology adoption. Perceived usefulness, which refers 
to “the degree to which a person believes that using a particular system would enhance his or her job 
performance” (Davis, 1989, p. 320), has been reported to be a robust predictor of technology adoption. In 
particular, perceived usefulness accounts for approximately 60% of intention to use (Venkatesh & Davis, 
2000). Given the importance of perceived usefulness, Venkatesh and Davis (2000) added two external 
processes to perceived usefulness in their extended TAM model (TAM2): social influence processes (e.g., 
subjective norms and personal image) and cognitive instrumental processes (e.g., job relevance, output 
quality, and result demonstrability) to improve its explanatory power (see Figure 1). 


Social influence processes indicate the influences of social forces (e.g., subjective norms and one’s personal 
image) on individuals’ perceived usefulness and intention to use new systems. Subjective norm refers to 
individuals’ perceptions of the expectations or pressure from important referents (Agudo-Peregrina et al., 
2014; Venkatesh & Davis, 2000). Personal image refers to individuals’ desire to portray a good image by 
following subjective norms to enhance membership as an inner circle member (Davis, 1989). 


In contrast, cognitive instrumental processes refer to individuals’ general cognitive processes, including job 
relevance, output quality, and result demonstrability, impacting their rational decisions about whether to 
adopt an innovation to maximize their utility (Venkatesh & Davis, 2000). Job relevance refers to “a function 
of the importance within one’s job of the set of tasks the system is capable of supporting” (Venkatesh & Davis 
2000, p. 191). Output quality indicates how effective it (flipped learning) is to obtain desirable learning 
outcomes or achieve expected results (Mitchell & Beach, 1976). Since social influence and cognitive 
instrumental processes influence perceived usefulness and intention to use, this study classified flipped 
learners using these two external processes and then compared the clusters in students’ perceptions, 
perceived usefulness, intention to register for flipped learning classes, and learning engagement. 


METHODS 


Context and Participants 


The participants of this study were 306 undergraduates who were enrolled in flipped learning classes in the 
2019 fall semester at a university in Korea. The survey was administered electronically using a Google survey. 
The survey link was passed on to students by nine professors teaching flipped learning courses. The survey 
was conducted two weeks prior to the end of the semester. The participants voluntarily completed the survey 
with no extra course credit (survey response rate: 68.45%) and there was no disadvantage for those who did 
not participate in the survey. 


More female students (N= 167, 54.6%) participated in this study than male students (N= 139, 45.4%). The 
distribution of the class year of the participants ranged from freshmen (N= 73, 23.9%), sophomores (N= 101, 
33.0%), juniors (N= 91, 29.7%), to seniors (N= 41, 13.4%). Since the survey was conducted across majors, the 
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Table 1. Research instruments 
Variables Number of items Cronbach’s alpha 


Subjective norms 2 .80 
Personal image 3 .80 
Relevance for learning 2 .87 
Quality of learning outcomes 2 89 
Perceived usefulness 4 96 
Registration intention 2 .96 


Learning engagement 10 .89 
Note: Adapted and modified from Venkatesh and Davis (2000), and Schreiner and Louis (2011) 


majors of participants varied widely, including energy resources (16.0%), nano-material science (15.7%), 
English language and literature (10.8%), education (10.1%), chemistry (10.1%), and others. 


Research Process and Measurement Instruments 


This study included two parts: (1) classifying the participants into meaningful clusters, and (2) making 
comparisons among the clusters for perceived usefulness, learning engagement, intention to register for 
flipped classes, and perceptions of flipped learning. 


The survey consisted of 32 questions, including demographic information (7 items) and questions measuring 
seven primary variables: subjective norms (2 items), personal image (3 items), relevance for learning (2 
items), the quality of learning outcomes (2 items), perceived usefulness (4 items), and intention to use (2 
items), and learning engagement (10 items). To measure the seven variables, we adapted and modified the 
TAM2 measurement (Venkatesh & Davis, 2000) except two items of voluntariness. Since taking flipped 
classes was not mandatory, voluntariness was not applicable to this study. Learning engagement was 
measured with 10 items of Schreiner and Louis’ (2011) scale. 


The survey questions were translated into Korean with reference to each original scale and then modified to 
make them more appropriate for a flipped learning context at a university in Korea (see Table 1). The survey 
was reviewed by two experts in the educational technology field. The participants were asked to answer each 
question ona 5-point Likert scale from 1 “strongly disagree” to 5 “strongly agree.” The participants were also 
asked an open-ended question about their perceptions of flipped classes. Table 1 describes the measurement 
scales of this study, including the number of items and the Cronbach’s alphas for the latent variables. The 
reliability coefficients for the latent variables were satisfactory given that Cronbach’s alpha for each latent 
variable was higher than .80. 


Data Analysis 


Prior to conducting the cluster analysis, we checked the normality assumption and the common method bias. 
Descriptive data including the means, standard deviations of the measurement variables, and correlations 
are shown in Table 2. The skewness and kurtosis of all variables ranged from -1 to 1; thus, the normal 
distribution assumption was acceptable (George & Mallery, 2010). 


Since the data of the current study were collected using one method (i.e., a survey), we performed Harman’s 
single-factor test to detect common method variance. The test results indicated that there was no common 
method variance because the total variance for a single factor was 39.25%, which is less than 50% (Podsakoff 
et al., 2003). 


To classify the participants into meaningful clusters, we conducted a cluster analysis using social influence 
(SI) and cognitive instrumental (Cl) processes. Cluster analysis refers to “a data reduction technique to reduce 
a large number of observations into a smaller number of groups” (Pastor, 2010, p. 41). Since SI and Cl 
processes are external factors of perceived usefulness and intention to use in TAM, we selected these 
variables for the cluster analysis. Since the number of clusters was unknown, we performed a two-step 
cluster analysis including (1) hierarchical cluster analysis and (2) (a non-hierarchical) K-means cluster analysis. 
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Table 2. Descriptive data 
Variables 1 2 3 4 5 6 7 


Subjective norms 1 74" 75" .67"" 67" 67" 70" 
Personal image 1 74" 62" .70"" 72" .70"" 
Relevance for learning 1 73" 74" 74" 71” 
Output quality 1 65" 66°" 63°" 
Perceived usefulness 1 87" ya 
Registration intention 1 74" 
Learning engagement 1 
Mean 3.31 3.16 3.60 3.69 3.88 3.85 3.36 
SD 99 90 1.01 1.02 98 1.09 81 
Skewness -.20 .09 -.43 -.48 -.72 -.83 .0O 
Kurtosis -.23 -.05 -.15 -.27 .08 .09 25 
Note: **p<.001, *p<.05 

Table 3. Final cluster centers for five clusters 

Cluster centers Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 
Subjective norms 4.76 3.42 1.93 3.26 2.21 
Personal image 4.36 3.42 2.13 2.97 2.20 
Relevance for learning 4.82 4.24 1.81 3.17 3.38 
Quality of learning results 4.90 4.36 1.74 3.19 3.41 


The purpose of hierarchical cluster analysis is to identity the number of clusters and cluster centers and the 
results are prerequisites for conducting K-means analysis (Aldenderfer & Balshfield, 1984). The hierarchical 
cluster analysis was performed using the Ward method based on squared Euclidean distance. Dendrogram, 
a visual representation showing the hierarchical relationships among cases, showed that five clusters were 
the most reasonable for the current data. Data analysis was performed using the statistical software SPSS 
(version 24.0). 


RESULTS 


K-means Cluster Analysis 


We conducted K-means cluster analysis based on the results of the hierarchical cluster analysis (i.e., the five- 
cluster solution). The number of cases in each cluster was as follows: cluster 1 (N= 53), cluster 2 (N= 83), 
cluster 3 (N= 29), cluster 4 (N= 103), and cluster 5 (N= 38). The final cluster centers of each cluster are 
summarized in Table 3. Cluster 1 has the highest means for the variables (i.e., the highest SI and Cl), while 
cluster 3 has the lowest means for the variables (i.e., the lowest SI and Cl). Cluster 2 has the second highest 
means for all variables (i.e., 2nd highest SI and Cl). Clusters 4 and 5 varied: cluster 4 scored higher on 
subjective norms and personal image (i.e., moderate SI and low Cl) than cluster 5. However, relevance for 
learning and quality of learning results were lower than cluster 5 (i.e., low SI and moderate Cl). 


The results of the analysis of variance (ANOVA) indicated that there were significant differences among the 
groups in the variables: (1) subjective norms, F(4, 301)= 186.228, p < .001; (2) personal image, F(4, 301)= 
117.804, p < .001; (3) relevance for learning, F(4, 301)= 227.764, p < .001); and (4) quality of learning results, 
F(4, 301)= 285.461, p < .001. The results of Scheffe’s post hoc multiple comparison test indicated that there 
were significant differences among the five clusters in subjective norms, personal image, relevance for 
learning, and quality of learning results. The final clusters are illustrated in Figure 2. 
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Table 4. Comparisons between clusters 
Clusters Pasiiee 
1 2 3 4 5 

Perceived usefulness 4.83 (.36) 4.46(.51) 2.15(.83)  3.64(.66) 3.27 (.69) 124.85" 
Registration intention 4.89(.28)  4.37(.76) 2.12(1.06)  3.57(.76) 3.36 (.99) 77.53" 
Learning engagement 4.26(.78)  3.66(.52)  2.25(.70)  3.12(.48) 2.98 (.48) 73.43” 


Note: ** p< .001, “p<.05 


Scheffe post hoc multiple comparison test (p-values) 
1&2 1&3 1&4 1&5 2&3 2&4 285 3&4 3&5 4&5 
Perceived usefulness .016 .000 .000 .000 .000 .000 .000 .000 .000 .037 
Registration intention .005 .000 .000 .000 .000 .000 .000 .000 .000 715 
Learning engagement .000 .000 .000 .000 .000 .000 .000 .000 .000 .788 


Differences in Perceived Usefulness, Intention to Use, and Learning Engagement in Flipped Learning 


We compared the five clusters in intention to register for flipped learning classes, perceived usefulness, and 
learning engagement in flipped learning using an analysis of variance (ANOVA). Cluster 1 had the highest 
scores on perceived usefulness, intention to register for flipped learning classes, and learning engagement 
whereas cluster 3 had the lowest scores on the three variables. Cluster 4 (moderate SI and low Cl) had higher 
scores on the three variables than cluster 5 (low SI and moderate Cl); however, there was statistical 
significance only between clusters 4 and 5 on perceived usefulness. Table 4 shows the results of the analysis 
of variance (ANOVA), indicating that there were significant differences among the five clusters in perceived 
usefulness (F(4, 301)= 124.85, p < .001), intention to register for flipped classes (F(4, 301)= 77.53, p < .001), 
and learning engagement (F(4, 301)= 73.43, p < .001). Scheffe’s post hoc test showed that there were also 
significant differences among clusters in perceived usefulness at the p < .05 level; however, no significant 
differences were found between clusters 4 and 5 in intention to register for flipped learning classes and 
learning engagement. 
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Table 5. Preference for flipped learning 


Cluster centers Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 
Positive 92.5% 100.0% 27.6% 74.8% 84.2% 
Negative 3.8% - 58.% 12.6% 13.2% 
Neutral or no answer 3.8% - 13.8% 12.6% 2.6% 


Table 6. Perceptions of flipped learning 


Cluster 
Category Sub-categories 1 2 3 4 5 Total 
(N=53) (N=83) (N=29) (N=103) (N=38) 
Online pre-learning materials 20 35 1 37 14 107 
Beacearnin Review opportunity for learning 7 7 0 10 4 28 
; 6 Individualized learning 3 3 0 6 2 14 
opportunities . : 
Self-directed learning 2 6 0 2 1 11 
Flexibility in learning time 3 7 0 5 2 17 
Question opportunities in class 2 0 0 1 0 3 
oe In-class discussion 2 6 0 6 3 17 
Class activities ; Sent ae, re 
Active participation in class activities (other 
: . i 1 4 1 2 1 9 
than discussion or questions) 
Understanding of lectures 7 14 1 9 2 33 
Retention of learning 2 2 1 0 0 5 
eine Critical thinking 0 0 0 0 1 1 
8 Efficiency of learning 0 (0) 0 2 4 6 
outcomes : . 
No learning effectiveness (0) 0 15 12 2 29 
Learning motivation 0 0 0 3 0 3 
Learning engagement and fun 0 4 0 il 2 7 
Amount of learning 2 3 1 2 1 9 
Others er , 
Excessive time for learning (0) 0 1 6) 5 
Total 51 91 24 99 39 304 


Students’ Perceptions of Flipped Learning 


Students’ perceptions of flipped learning as reported in open-ended responses were analyzed using NVivo 
12, qualitative data analyses software. Among the 304 comments about the perceptions of flipped learning, 
78.6% were positive and 12.1% were negative (9.3% were neutral or missing values). Each cluster revealed 
different perceptions of flipped learning and there were significant differences among the five clusters (x2(4) 
= 80.321, p < .001), as shown in Table 5. 


The Signature of Flipped Learning: Online Pre-Learning Sessions 


The author performed pilot coding of verbatim responses and generated four categories (i.e., pre-learning 
materials, class activities, learning outcomes, and others) and 17 sub-categories (see Table 6). Since coding 
was completed by a single researcher, the coding was done twice with a time lapse interval of one week to 
ensure reliability. This iterative process enabled the researchers to develop mutually exclusive and 
conceptually congruent codes (Merriam, 1998). Misplaced and double-meaning codes were revised through 
this iterative process. The cluster comparisons of the perceptions of flipped learning are illustrated in Table 
6. 


The most frequent comments about the perceptions of flipped learning were “using pre-learning online 
materials” (N=107, 35.2%). Sample verbatim statements include, “Online pre-learning materials help me 
understand the class more effectively” (Cluster 1, female/junior/Energy engineering), and “It is good to know 
in advance what | will learn in class due to online pre-learning materials’ (Cluster 2, 
female/sophomore/Fashion design). Review opportunities for learning were also mentioned as the strength 
of flipped learning, such as “Flipped learning allows me to repeat the pre-learning materials until | 
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understand” (Cluster 1, female/junior/Education), and “It gives me learning opportunities to understand 
what | missed in class” (Cluster 4, male/senior/Nano science). A freshman student elaborated further on her 
response: 


Once | miss a lecture in class, there is no way to do damage control and it negatively 
affects the remaining classes. Or sometimes, | cannot concentrate on lectures because of 
my physical condition. However, flipped learning allows me to study wherever | am, such 
as at home or even on the subway and to re-play and repeat as many times as | want. 
That’s why | think that flipped learning is a more effective instructional approach than 
lectures. 


(Cluster 1/female/freshman/Education). 


Effectiveness and efficiency of flipped learning: Learning outcomes and class activities 


There were also comments about the strengths of flipped learning including individualization and self- 
directed learning. Sample statements include, “I can selectively focus on what | deem to be important and 
skip the remaining parts” (Cluster 4, female/freshman/Education), “The strength of flipped learning is that | 
can repeatedly play pre-learning materials at my own pace. | like listening to the online lectures over and 
over again at a faster speed and flipped learning fits my needs. Since the online materials remain on 
Blackboard, | can look up the part | don’t understand in the lecture after class” (Cluster 1, 
female/freshmen/Chemistry) [Individualization], and “Flipped learning is an appropriate instructional 
approach to facilitate students’ own understanding of learning materials and problem-solving skills” (Cluster 
2, male/freshman/Chemistry) [Self-directed learning]. 


As for class activities, three sub-categories emerged: (1) question opportunities, (2) in-depth discussion, and 
(3) active participation in other class activities. In-depth discussion was mentioned the most (N= 17): “In 
flipped learning, | can discuss about an issue with others who have different perspectives” (Cluster 2, 
male/junior/Hospitality), and “Flipped learning is effective because | can share what | learned from pre- 
learning online materials with classmates in the discussion” (Cluster 4, female/sophomore/Education). 


Many participants reported on the learning effectiveness of flipped learning (N= 55), such as facilitating 
“understanding of lectures” (N= 33), “efficiency of learning” (N= 6), and “retention of learning” (N= 5). Sample 
statements include: 


Sometimes cognitive overload occurs in class because students are required to 
immediately understand what they just (or concurrently) learned. Flipped learning allows 
students to understand the learning content at their own pace. It also includes many class 
activities, not just lecturing, so it helps understanding of learning (Cluster 4, 
female/senior/Economics). 


Another statement is “Since online pre-learning materials facilitate understanding of lectures, it helps 
retention of learning without additional efforts for learning” (Cluster2, male/senior/Physical education). 


Comments about learning motivation (N= 3), fun, and learning engagement (N= 7), include, “In flipped 
learning, | have clear learning motivation to participate in class activities” (Cluster 4, male/sophomore/Energy 
engineering), “[Flipped learning is] less boring than lectures” (Cluster 5, male/senior/Nano science), and “The 
retention of learning improved in flipped learning because it is fun” (Cluster 5, female/sophomore/Media 
communication). 


Negative perspectives on flipped learning 


There were also negative comments about the effectiveness of flipped learning (N=29). Most of the negative 
comments were from students in clusters 3 and 4. Sample statements include: 
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Flipped learning is not effective for me. The online video materials are too lengthy. | don’t 
see a connection between the video materials and lectures. | prefer succinct lectures in 
class. Many students play the online video materials at 2x speed or listen to them at one 
sitting, so it does not help (Cluster 3, female/sophomore/Music). 


and “I don’t see the big difference [between flipped learning and traditional lectures]” (Cluster 4, 
female/senior/Animation). There were also complaints about the required learning time for flipped learning 
online materials, such as “Flipped learning requires excessive preparation time besides class hours” (Cluster 
3, male/junior/English). 


DISCUSSION 


This study investigated flipped learners’ perceptions, perceived use, intention to register for flipped learning 
classes, and learning engagement using cluster analysis. We classified 306 flipped learners into five clusters 
based on their level of social influence (SI) and cognitive instrumental (Cl) processes using TAM2 (Venkatesh 
& Davis, 2000). The results of cluster analysis yielded the following combinations: cluster 1 (N= 53, highest SI 
and Cl), cluster 2 (N= 83, the second highest SI and Cl), cluster 3 (N= 29, lowest SI and Cl), cluster 4 (N= 103, 
moderate SI and low Cl), and cluster 5 (N= 38, low SI and moderate Cl). There were statistically significant 
differences among the five clusters in perceived usefulness, intention to register for flipped learning classes, 
and learning engagement. These differences confirmed the validity of the cluster classification using social 
influence and cognitive instrumental processes. The results of a post hoc test indicated that those who had 
moderate the social influence process with a low cognitive instrumental process (cluster 4) had statistically 
higher perceived usefulness for flipped learning than students who had low social influence with a moderate 
cognitive instrumental process (cluster 5). That is, those who perceived that their professors, parents, or 
friends had high expectations of them (subjective norm) and those who wanted to portray a good image in 
class or at school (personal image) were likely to have a higher level of perceived usefulness of flipped 
learning than students who understood the relevance of flipped learning and expected successful learning 
outcomes in flipped learning classes. However, there were no significant differences between clusters 4 and 
5 in intention to register for flipped learning classes and learning engagement in flipped learning. For practical 
implications, instructors and school administrators should consider the social influence process and cognitive 
instrumental process together to predict students’ intention to register for flipped classes and learning 
engagement. 


In this study, 78.6% of the participants had a positive perception of flipped learning. This is higher than Hao’s 
(2016) survey results (roughly 60%). Of the 306 participants, 196 participants (64.1%) stated that they 
intended to take flipped learning classes in the future, which is also higher than Hao’s study (44%), and Forsey 
et al.’s (2013) study (53%). The discrepancies between students’ general perception and the intention to take 
flipped learning classes indicate that there are still areas for improvement in flipped learning classes. It is 
necessary to investigate why those who were positive about flipped learning were also reluctant to take 
flipped learning classes in the future. 


Students’ perceptions of and preferences for flipped learning were statistically different among the five 
clusters. Most participants in clusters 1 (highest SI and Cl) and 2 (second highest SI and Cl) were positive 
about flipped learning as an instructional approach (92.5% and 100% respectively). However, 58% of cluster 
3 (lowest SI and Cl) were negative about flipped learning and only 27.6% were positive. This finding indicates 
that the social influence process and the cognitive instrumental process of TAM are valid predictors of 
students’ perceptions of flipped learning. Cluster 5 (low SI and moderate Cl) was more positive about flipped 
learning (84.2%) than cluster 4 (moderate SI and low Cl) (74.8%). However, there were no significant 
differences in the negative perceptions of flipped learning between clusters 4 (12.6%) and 5 (13.2%). This 
result indicates that there is a wide range of student perceptions of flipped learning, and it is necessary to 
provide customized learning support and implement instructional strategies for each type of flipped learners. 
Given that there were no significant differences in students’ perceptions and preferences between the 
cluster with a high social influence process and low cognitive instrumental process and the cluster with 
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reversed features (i.e., vice versa), the social influence process and cognitive instrumental process must be 
considered together instead of separately for use in cluster analysis. 


We analyzed the participants’ open-ended comments about their perceptions of flipped learning, such as 
why they liked or disliked flipped learning (N= 304). The following four themes emerged: (1) online pre- 
learning opportunities, (2) class activities, (3) learning outcomes, (4) other (e.g., workload and learning 
times). The most frequent comments about flipped learnng was online pre-learning sessions. This result 
confirmed previous research findings that have emphasized the importance of pre-learning sessions in 
flipped learning by Lee and Choi (2019), Lo and Hew (2017), and Steen-Utheim and Foldnes (2018). An online 
pre-learning session must appear to be a distinctive feature to students. Students recognized that these 
sessions are useful for their learning because it makes it possible to “facilitate understanding of lectures,” 
“review opportunities for learning,” “[provide] flexible learning time,” “individualized learning,” and 
“meaningful in-class discussions.” As Lee and Choi (2019), and Lo and Hew (2017) emphasized, providing 
students with high-quality pre-session learning materials is critical to the success of flipped learning. 
Considering the importance of pre-learning sessions in a flipped learning class, instructors should encourage 
students to complete the pre-learning sessions and school administrators should provide those who teach 
flipped classes with sufficient support to produce high quality pre-learning materials. 


It is noticeable that affective domains, such as learning motivation, learning engagement, and fun, were 
included in the perceptions of flipped learning as a strength of flipped learning. These advantages are similar 
to the research findings by Steen-Utheim and Foldnes (2018), Hao (2016), Nouri (2016), and Forsey et al. 
(2013). Hao’s (2016) research included “recognition of learning needs,” “more interaction in classes,” and 
“more effective use of class time,” and Nouri’s (2016) conclusions, included “flexibility and mobility of 
learning,” “support for learning,” and “meaningful learning activities.” Our research findings also 
substantially overlap with Forsey et al.’s (2013) observations, including “flexibility,” “richness of learning 
contents,” and “productivity in learning,” and Wang and Zhu’s (2019) findings about flipped class students’ 
preferences for student-student interaction and active learning results in class. This finding indicates that 
students recognize flipped learning as an instructional approach that they can have fun with and that it can 
enhance learning motivation and learning engagement. Steen-Utheim and Foldnes (2018) emphasized the 
affective aspects of learning engagement in flipped learning, stating that it is necessary to make flipped 
learning experiences more meaningful by making efforts to increase learning motivation, learning 
engagement, and even fun. 


The students also had negative perceptions of flipped learning, including “no learning effectiveness” and 
“excessive time for learning.” This finding supported the research results by He et al. (2016), Forsey et al. 
(2013), and Hao (2016). Even though students’ perceptions of flipped learning were largely positive, He et al. 
(2016) explained that there is a wide spectrum of students’ perception of flipped learning varies from positive 
to negative. Forsey et al. (2013) indicated that students were anxious about replacing instructor-led class 
lectures with online pre-learning materials. Finally, Hao (2016) found that slight less than 40% of flipped 
learners reported that flipped learning satisfied with their learning needs, indicating that students disliked 
the “requirement self-discipline,” “requirement of prior knowledge and academic motivation,” and “less 
learner control” in flipped learning. 


In this study, we found that participants’ perceptions of flipped learning are different by cluster. There was 
no single negative comment from cluster 1 (highest SI and Cl) or cluster 2 (second highest SI and Cl). Negative 
comments, such as “no learning effectiveness” and “excessive time for learning” mostly came from students 
in cluster 3 (lowest SI and Cl). Interestingly, there were more negative comments from students in cluster 4 
(moderate SI and low Cl) than cluster 5 (low SI and moderate Cl). These findings confirm Nouri’s (2006) 
research that different flipped learner groups have different perceptions of flipped learning. 


There are two plausible explanations about the discrepancies between the previous literature and the 
current study. First, most negative comments came from the group with low social influence and cognitive 
instrumental process and their perceived usefulness, intention to register for flipped learning classes, and 
learning engagement were the lowest of all the clusters. Hence, this finding should be interpreted considering 
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the research design (i.e., cluster analysis). The other reason may be that these perceptions may also be 
influenced by how instructors implement flipped learning, such as the flipped class policies or grading system. 
For example, Wang and Zhu (2019) explained that the quality of pre-learning materials is an influencing factor 
of students’ satisfaction level of MOOCs. 


Therefore, it is necessary to identify meaningful variables to segment flipped learners so instructors can 
implement effective, practical, and feasible instructional strategies and provide appropriate learning support 
to attain learning outcomes. 


Limitations and Recommendations for Future Research 


This study has several limitations. It is important to point out these limitations for those who are interested 
in extending the current study. We used cluster analysis as an analytic method in this study. The results of 
cluster analysis may change depending on which variables are used to classify the clusters (e.g., SI and Cl 
processes in the current study, learning achievement, or social economic status). If future researchers select 
different variables to classify respondents using cluster analysis, the results may be different. Therefore, it is 
important to choose reasonable and meaningful variables to select and classify respondents considering the 
research purpose. Cluster analysis is a practical method to classify a large number of respondents with 
homogenous characteristics into meaningful clusters. However, Pastor (2010) explained that cluster analysis 
could yield clusters by creating a structure even if the structure does not actually exist (e.g., exploratory 
factor analysis). While the validity of the results of exploratory factor analysis can be examined using 
confirmatory factor analysis, cluster analysis does not have a specific analytic technique to validate the 
results. Thus, it can be validated using an analysis of variance (ANOVA) or chi square tests with different 
variables. 


Another limitation of this study is the limited generalizability since this study was conducted with 
undergraduate students who were enrolled in flipped learning classes in Korea. Future studies are 
encouraged to expand the scope with participants from different countries or in multicultural contexts. In 
addition, we included an open-ended question in the survey to investigate the participants’ perceptions of 
flipped learning. Future researchers who extend the current research should collect more qualitative data, 
such as focus group interviews or individual in-depth interviews to obtain robust, rich, and vivid research 
findings. He et al. (2016) criticized many flipped learning studies that were conducted without proper controls 
to validate the effects of treatment. This study also did not have a comparison group (e.g., those who were 
enrolled in the same courses delivered in traditional classroom) for a practical reason. Future studies should 
use a solid research method and control extraneous variables to validate the research findings in this study. 
Finally, given that the current study is limited to examining the differences among the clusters in perceived 
usefulness, learning engagement, perceptions of flipped learning, and intention to register for flipped 
learning classes, future studies are strongly encouraged to expand the scope of this study by examining group 
differences in academic achievement or learning outcomes. 


CONCLUSIONS 


Flipped learning is a popular instructional approach in higher education. A frequently emphasized strength 
of flipped learning is the cost-effectiveness and efficiency to implement a learner-cantered approach to a 
large number of students. However, students have unique learner characteristics and needs. We cannot 
ignore these individual differences for educational purposes, but we also cannot completely customize 
programs for every single student. It is necessary to compromise between the ideal and the possible. If we 
identify critical factors to classify meaningful learner groups in flipped learning, it may enable us to provide 
students with significant and feasible customized learning support. The phrase “one size does not fit all” 
applies to flipped learning. 
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